Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)

Reply to this post | Go Back
View Post [edit]

Poster: Freestone Slice Date: Feb 15, 2014 1:18am
Forum: forums Subject: Why does the WayBack Machine not crawl LiveJournal?

There are actually a couple of sites like this, but LJ is the most recent one I've noticed, so I'm using it as an example. It does have a robots.txt file, but the file doesn't exclude archive.org either specifically or as a blanket restriction. What's up?

Terms of Use (10 Mar 2001)